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1.0  Introduction 


The  Team-Based  Assessment  of  Soeio-Teehnical  Logistics  (TASL)  research 
program  was  sponsored  by  the  Air  Force  Research  Laboratory’s  Logistics  Readiness 
Branch  (AFRL/RHAL)  under  the  Technology  for  Agile  Combat  Support  (TAGS) 
contract  (FA  8650-D-6546,  Delivery  Order  #3).  The  TASL  research  was  accomplished 
during  the  period  of  10  June  2005  to  9  April  2008.  The  purpose  of  the  TASL  program 
was  to  research  and  develop  methodologies,  frameworks,  and  metrics  that  in  turn,  could 
be  applied  to  support  the  evaluation  of  Soeio-Teehnical  Systems  (STS)  and  Computer 
Supported  Cooperative  Work  (CSCW)  technologies.  The  overarching  goals  of  the  TASL 
program  included  the  following: 

•  Analyzing  collaborative  systems  and  technologies  from  a  socio-technical 
perspective  for  the  purpose  of: 

o  Developing  an  in-house,  socio-technical  research  capability  to  assess  and 
inform  the  design,  development,  and  implementation  of  collaborative 
systems  in  military  planning  and  command  and  control  environments 
(particularly  logistics). 

o  Performing  field  research  in  selected  environments  to  inform  the  design 
and  execution  of  controlled,  in-house  experimentation  activities  to 
examine  various  aspects  of  collaboration  (e.g.,  communication  modes, 
team  roles,  etc.). 

•  Acquiring  a  better  understanding  of  the  social  context  of  work  in  real-world 
STS  environments. 

The  primary  objective  of  the  TASL  research  effort  was  to  improve  warfighter 
team  performance  in  distributed,  operational  environments  involving  collaborative 
logistics  activities  such  as  crisis  action  planning,  dynamic  re -planning,  and  command  and 
control.  In  support  of  this  objective,  research  goals  were  focused  on  improving  human 
collaboration  in  crisis  action  planning  and/or  command  and  control  environments,  and 
expanding  our  knowledge  and  understanding  of  the  impact  of  collaborative  systems  on 
human  (team)  behavior  and  performance. 
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1.1  Background 


As  the  need  for  eommunieation  and  eollaboration  among  distributed  teams  has 
inereased,  the  world  has  seen  an  explosion  in  software  and  hardware  teehnologies  (e.g., 
web  eonfereneing,  shared  whiteboards,  ete.)  intended  to  support  a  range  of  eomplex  work 
aetivities  related  to  deeision  making,  planning,  problem  solving,  developing  eourses  of 
aetion,  ete.  Systems  engineering  design  methodologies  used  to  develop  traditional 
eollaborative  systems  have  not  kept  paee  with  the  inereasingly  eomplex  eollaboration 
required  in  today’s  military.  Transaetion  based  proeessing  systems  that  were  primarily 
designed  to  support  single  users  interfaeing  with  a  computer  -  not  the  collaboration 
among  multiple  users  and  systems  -  are  no  longer  sufficient.  Potential  users  of  these 
technologies  and  systems  can  find  the  technical  options  overwhelming  and  difficult  to 
compare.  In  fact,  few  methodologies,  frameworks,  or  metrics  exist  to  evaluate  how 
individual  and  team  performance  is  impacted  by  the  introduction  of  collaborative 
technologies  and  systems  in  organizations.  As  a  result,  technologies  are  often  purchased 
and  placed  within  an  existing  organizational  structure  with  little  understanding  of  how  the 
technology  changes  the  way  work  gets  done  and  how  it  will  support  or  hinder  a  team  (and 
organization)  in  accomplishing  its  goals.  The  emergence  of  ubiquitous  computing  as  the 
next  wave  of  organizational  computing  offers  new  possibilities  and  opportunities  for 
organizations  to  improve  their  productivity  and  effectiveness.  However,  we  need  a  better 
understanding  of  how  these  technologies  affect  organizations  and  the  social  context  of  the 
work  for  which  collaborative  systems  are  intended  to  support.  The  “social”  context 
includes  organizational  culture,  team  and  group  dynamics,  individual  personalities,  etc. 
(Rogers  and  Bellotti,  1997). 

Very  detailed  computing  architectures  have  been  published  that  emphasize  the 
information  technology  (IT)  aspects  (i.e.,  system  interoperability,  networking, 
information  assurance,  etc.)  associated  with  collaboration  technologies.  In  fact,  because 
of  the  growing  interest  in  leveraging  collaboration  technologies  to  support  distributed 
operations,  the  Department  of  Defense  (DoD)  has  given  considerable  attention  to  the 
testing  of  collaboration  technologies  through  programs  such  as  the  DoD  Defense 
Collaboration  Tool  Suite  (DCTS)  managed  by  the  Defense  Information  Systems  Agency 
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(DISA).  However,  the  emphasis  of  the  testing  assoeiated  with  these  teehnologies  or 
applieations  is  primarily  from  an  IT  perspeetive  (i.e.,  seeurity  and  interoperability) 
against  well-defined  requirements  or  standards.  What  is  lacking  is  a  full  consideration  of 
the  human  aspects  of  system  operation,  and  in  particular  how  collaborative  technologies 
or  CSCW  applications  improve  or  impede  human  performance  in  distributed 
environments  involving  collaboration  between  warfighters.  The  powerful  networked 
computing  environments  supporting  the  implementation  of  collaborative  technologies 
and  applications  will  not  reach  their  full  potential  without  explicit  consideration  of  the 
human-centric  element.  The  TASL  program  focused  on  addressing  this  gap  by 
researching  and  making  progress  toward  developing  a  framework  for  assessing 
collaborative  technologies  and  teams  from  a  human  and  organizational  perspective, 
recognizing  that  simply  testing  collaborative  technologies  and  systems 
(hardware/sofiware)  from  a  reliability  and  network  connectivity  standpoint  does  not 
guarantee  successful  implementation. 

1.2  Terminology 

1.2.1  Socio-Technical  System  (STS) 


A  Socio-Technical  System  (STS)  theory  views  work  organizations  as  comprised 
of  two  interdependent  subsystems  -  a  technical  subsystem  and  a  social  subsystem,  as 
depicted  in  Figure  1 . 


Figure  1.  Socio-Technical  System  Perspective 
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The  technical  subsystem  includes  the  operating  policies  and  procedures,  business 
methods,  and  task  procedures  guiding  the  accomplishment  of  work,  as  well  as  the 
information  technology  and  software  applications  supporting  the  same.  The  social 
subsystem  encompasses  the  individuals  working  in  the  organization,  including  the 
knowledge,  attitudes,  values  and  needs  they  bring  to  the  work  environment,  as  well  as  the 
organizational  culture,  power  hierarchy,  and  reward  systems  within  the  organization. 
These  subsystems  have  a  direct  impact  on  business  processes  and  are  therefore  a  key  area 
of  interest  in  STS  design  practices  that  attempt  to  take  into  account  the  network  of  users, 
developers,  information  technologies,  and  the  environments  in  which  a  system  will  be 
used  and  supported.  The  STS  design  process  includes  the  design  of  the  human-computer 
interface  and  patterns  of  human-computer  interaction.  It  stands  in  opposition  to 
traditional  system  or  software  engineering  design  methods  that  focus  attention 
exclusively  or  primarily  on  the  activities  of  system  engineers  who  design  the 
computational  functions  and  features  of  a  new  system,  and  who  use  computer-aided 
design  tools  and  techniques  such  as  the  Unified  Modeling  Language  (UML)  to  capture, 
formalize,  and  portray  the  results  of  such  a  design  process.  In  contrast,  STS  design  is 
concerned  with  advocacy  of  the  direct  participation  of  end-users  in  the  information 
system  design  process  (Scacchi,  2004).  Some  examples  of  socio-technical  systems 
include  emergency  response  systems,  logistics  planning  systems,  remote  medicine,  and 
unmanned  aerial  vehicle  ground  control  systems. 

1.2.2  Computer  Supported  Cooperative  Work  (CSCW) 

Computer  Supported  Cooperative  Work  (CSCW)  refers  to  the  theoretical 
foundations  and  methodologies  for  teamwork  and  corresponding  computer  support.  It  is 
concerned  with  how  collaborative  activities  and  their  coordination  can  be  best  supported 
by  means  of  computer  systems  (Carstensen  and  Schmidt,  2003).  CSCW  attempts  to 
combine  the  understanding  of  the  way  people  work  in  groups  with  the  enabling 
technologies  of  computer  networking,  and  associated  hardware,  software,  services,  and 
techniques  (Borghoff  and  Schlicter,  2000).  In  some  circles,  CSCW  is  used 
synonymously  with  the  term  “groupware”;  however,  the  latter  is  attributed  more 
frequently  to  the  underlying  technologies  or  tools  (e.g.,  instant  messaging,  chat. 
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whiteboards)  and  other  practical  solutions  that  support  the  collaborative  work  of  groups 
and  teams. 

With  regard  to  CSCW,  the  term  “work”  refers  to  the  work  system  and  associated 
components  depicted  in  the  Leavitt  Rhombus  in  Figure  2,  as  well  as  the  interactions 
between  these  components.  Therefore,  from  a  CSCW  perspective,  consideration  of  each 
of  the  four  work  system  components  (technology,  people,  process,  and  organization),  and 
their  interactions,  is  critical  to  the  design  of  any  collaborative  system. 


Figure  2,  Work  System  Components  -  Leavitt  Rhombus 


1.2.3  Team 

A  team  can  be  defined  as  a  distinguishable  set  of  two  or  more  people  that:  1) 
interact  dynamically,  interdependently,  and  adaptively  toward  a  common  goal  and  valued 
goal/objective/mission;  2)  have  specific  roles  or  functions  to  perform;  and  3)  have  a 
limited  life  span  of  membership  (Salas,  Dickinson,  Converse,  and  Tannenbaum,  1992).  It 
should  be  noted  that  the  characteristics  of  “interdependence”  and  “limited  life  span”,  are 
arguably  two  key  differentiators  between  teams  and  groups. 

2.0  Study  Approach  and  Analysis 

The  research  conducted  as  part  of  the  TASL  program  proceeded  along  three 
distinct,  yet  complementary  vectors,  all  focused  on  supporting  an  analysis  of 
collaborative  systems  and  technologies  from  a  socio-technical  perspective.  These  vectors 
included:  1)  a  review  and  assessment  of  “human  centered”  research  areas  related  to 
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collaborative  systems;  2)  the  development  of  a  framework  to  support  the  assessment  and 
evaluation  of  eollaboration  and  teamwork  in  organizations  (ineluding  data  eolleetion  in  a 
relevant  environment  to  inform  initial  design  and  subsequent  refinements  to  the 
framework);  and  3)  investigating  the  use  of  organizational  simulation  to  support  the 
design  and  assessment  of  organizational  struetures  in  command  and  control 
environments. 

2.1  Literature  Search  and  Analysis  of  Potential  Research  Areas 

The  TASL  team  was  eomprised  of  personnel  (eontraetor  and  government) 
representing  several  aeademie  diseiplines  ineluding  eognitive  psyehology,  industrial  and 
organizational  psyehology,  eomputer  science,  human  factors  engineering,  and  logistics. 
The  eomposition  of  the  team  allowed  us  to  take  a  multi-diseipline  approaeh  to  the  task  of 
assessing  and  formulating  potential  researeh  areas  related  to  the  design,  development,  and 
evaluation  of  eollaborative  systems  in  logisties  eommand  and  eontrol  environments  from 
a  soeio-teohnieal  perspeetive.  In  performing  the  literature  seareh  and  analysis  of 
potential  researeh  areas,  our  goals  were  to  aequire  a  better  understanding  of  eurrent 
methods,  tools,  and  measures  that  might  be  leveraged  to  support  a  soeio-teohnieal 
analysis  of  eollaborative  systems  and  teohnologies,  and  to  identify  potential  gaps  where 
additional  research  was  needed.  In  approaching  this  task,  a  series  of  workshops  were 
planned  and  oonduoted  that  brought  together  members  of  the  TASL  team  representing  the 
areas  of  expertise  identified  above  (Appendix  A). 

During  the  first  workshop,  the  focus  was  on  developing  some  oommon  ground 
and  understanding  with  respeot  to  the  objeotives  and  goals  of  the  TASL  researeh 
program,  bounding  the  “problem  spaee”  to  support  the  discussions  leading  to  the 
formulation  of  potential  researeh.  The  model  shown  in  Figure  3  was  derived  to  more 
elearly  delineate  the  seope  and  objeetives  of  the  TASL  researeh  program  and  to  help 
formulate  speeifie  foeus  areas  related  to  the  design,  development,  and  deployment  of 
eollaborative  systems  to  be  explored  further  in  the  workshops  that  followed. 
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Figure  3,  Model  for  Assessment  of  Potential  TASL  Research  Areas 

During  the  second  workshop,  the  TASL  team  focused  attention  on  discussing  and 
developing  potential  areas  of  interest  and  related  questions  corresponding  to  one  or  more 
of  the  model  components  depicted  in  Figure  3  (e.g.,  “metrics”  is  directly  related  to  the 
system  development  phase  encompassing  “test  and  evaluation”).  The  analysis  and 
discussions  of  each  of  these  components,  as  well  as  consideration  of  the  skills  and 
capabilities  of  TASL  team  members,  resulted  in  the  formulation  of  four  specific,  “human- 
centered”  areas  of  potential  research  that  included  the  following; 

•  Organization  /  Process  /  Culture 

o  Barriers  to  Cross-Agency  (Cross-Functional)  Collaboration 

o  Dimensions  of  Organizational  and  Technology  Influences  on 

Collaboration 

•  Training 

o  Impact  of  Collaborative  Systems  on  Training 

•  Evaluation  and  Assessment 

o  Assessment  of  Collaboration  in  Logistics  Planning  -  Methods  and 
Measures 

•  Design  and  Visualization 

o  Construction  of  a  Common  Operating  Picture  in  Dynamically  Changing 
Environments 

The  specific  research  questions  discussed  and  developed  for  each  area  above  are 
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included  in  Appendix  B.  A  team  lead  and  supporting  members  (based  on  skills  and 
interest)  were  assigned  to  each  of  the  areas  identified  above  and  tasked  with  eonducting  a 
more  thorough  literature  review  and  investigation  of  their  respective  areas.  This  included 
eompleting  a  more  thorough  domain  analysis  and  assoeiated  literature  search  activities 
for  their  respective  areas,  and  developing  a  plan  that  outlined  proposed  researeh  efforts 
that  might  be  pursued  under  the  TASL  program.  In  developing  these  research  plans, 
teams  were  advised  to  focus  attention  on  collaborative  planning  and/or  eommand  and 
eontrol  systems  in  an  Air  Foree  or  joint  logistics  environment. 

A  final  workshop  was  convened  to  bring  together  team  members  representing  the 
four  topic  areas  so  they  could  present  the  results  of  their  literature  search  and  domain 
analysis  activities,  as  well  outline  and  discuss  proposed  topics  for  further  research  and 
investigation  intended  to  address  the  questions  posed  for  their  respeetive  areas  (see 
Appendix  B).  A  summary  of  the  results  of  the  literature  search  and  analysis  for  each  of 
these  areas  identified  above  is  included  below,  and  presented  in  more  detail  in  separate 
reports  that  were  developed  and  delivered  to  AFRL/RHAL  (see  Appendix  B  for  report 
referenees). 

2,1.1  Organization  /  Process  /  Culture 

In  addressing  this  area,  we  were  concerned  with  gaining  a  better  understanding  of 
previous  and  on-going  researeh  related  to  eross-agency  collaboration  (particularly  virtual 
collaboration  in  Air  Force  or  other  military  domains)  with  an  emphasis  on  a)  identifying 
barriers  or  impediments  to  collaboration,  and  b)  the  infiuenees  of  teehnology  and 
organization  on  collaboration.  With  respect  to  the  former,  we  discovered  that  there  was 
little,  if  any  researeh  directly  addressing  eross-agency  collaboration  in  Air  Force 
command  and  control  and  logistics  environments.  In  faet,  the  topie  of  eross-agency  or 
interagency  collaboration  was  most  frequently  discussed  in  the  area  of  public  health  via 
analysis  of  case  studies.  The  literature  seareh  did  reveal  a  signifieant  number  of  journal 
artieles,  and  teehnieal  papers  related  to  collaborative  technologies  and  the  impact  on 
teamwork,  including  Wainfan  and  Davis  (2004),  who  investigated  the  topie  of  virtual 
collaboration  and  identified  potential  barriers  or  problems  in  eollaboration  based  on  the 
medium  involved  (faee-to-face,  video  conferencing,  audio  conferencing,  or  computer 
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mediation  communication).  Based  on  the  work  of  Wain  fan  and  Davis,  as  well  as  our 
review  of  other  sources,  we  identified  some  key  benefits  and  barriers  to  collaboration  that 
might  be  of  interest  to  the  TASL  program  (see  Figure  4). 


Benefits  of  Virtual 

Barriers  to  Virtual 

Collaboration 

Collaboration 

•  Greater  equality  of 

•  Information  loss 

participation 

•  Depersonalization 

•  Broadening  reach 

•  Reduced  cohesion 

•  Rapid  response  time 

•  Reduced  participation 

•  Adaptability 

•  Leadership  emergence 

•  Time  and  money 

suppressed 

•  Meets  “real-world” 

•  Adversarial  local 

demands 

coalitions 

•  Better  than  FTF  for 

•  More  extreme  decisions  - 

generative  tasks 

Risky  Shift 

•  Disinhibition  -  “Flaming” 

•  Delayed,  more  fragile 
trust 

•  Increased  cognitive  load 

Figure  4,  Benefits  and  Barriers  to  Virtual  Collaboration 


It  is  important  to  note  that  some  of  the  benefits  and  barriers  identified  in  Figure  4 
may  be  situation  specific.  For  example,  greater  equality  of  participation  is  touted  as  a 
benefit  of  virtual  collaboration  in  that  computer-mediated  collaboration  allows  one  to 
contribute  to  the  conversation  without  interrupting  others.  Further,  authority  is  less 
visible  in  computer-mediated  collaboration,  often  reducing  inhibitions.  As  a  result,  more 
ideas  are  generated  and  more  team  members  find  a  voice  in  the  conversation.  However, 
there  may  be  circumstances  in  which  this  open  dialog  is  unwelcome  and  even  a  hindrance 
to  decision  making  or  planning.  Therefore,  it  was  determined  that  it  would  be  important 
to  consider  the  context  in  which  the  original  research  was  conducted,  and  to  perhaps 
prioritize  those  elements  that  seem  most  likely  to  be  substantial  benefits  or  barriers  in  the 
context  of  TASL  program. 

In  investigating  the  influences  of  technology  and  organization  on  collaboration,  a 
fairly  broad  literature  review  did  not  address  the  issues  (and  associated  impact)  that 
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Shalin,  Bass,  and  Wales  (2005)  have  observed  in  real  government  settings.  Teehnology 
needs  analysis  did  not  appear  to  be  prineipled — haphazard  at  best,  and  politieally  driven 
at  worst.  In  this  case,  possibly  developing  a  more  objective  foundation  (and 
corresponding  culture)  for  identifying  useful  technology  could  promote  a  more  efficient 
and  effective  use  of  resources. 

The  impact  of  centralized  and  decentralized  software  development  practices  on 
the  efficiency  of  technology  development  versus  resulting  work  practice  needs  to  be 
addressed.  Efficient  technology  development  can  result  in  an  inefficient  work  practice 
rife  with  workarounds,  which  are  often  hidden  to  the  outside  observer.  In  some  cases,  the 
management  that  makes  decisions  is  not  sufficiently  engaged  in  the  current  demands  of 
the  task  environment,  and  primarily  rewarded  for  software  delivery  that  is  within  budget 
rather  than  helpful  for  the  real  challenges.  Tinkering  with  interfaces  and  broadcasting 
systems  -  the  sorts  of  interventions  one  finds  in  the  literature  -  are  likely  irrelevant  in  the 
face  of  these  much  more  pervasive  influences.  In  addition,  they  cannot  be  understood  by 
examining  behavior  in  the  two-hour  laboratory  experiment.  It  was  concluded  that  long 
term,  integrative  research  would  be  required  to  provide  a  much  more  principled 
foundation,  with  quantifiable  consequences,  for  the  design  of  technology  to  support 
collaborative  work. 

2,1.2  Training 

The  focus  of  our  literature  search  and  investigation  pertaining  to  the  area  of 
training  was  on  the  impact  of  collaborative  systems  on  training  from  a  socio-technical 
perspective  -  primarily  the  impact  on  the  design,  development,  delivery,  and  evaluation 
of  training  supporting  teams  using  collaborative  systems.  Our  review  of  the  literature 
revealed  extensive  but  largely  non-overlapping  lines  of  research.  We  found  substantial 
research  on  teams — their  function  and  processes  as  well  as  outcomes  of  teams  (both 
affective  and  performance),  but  much  of  the  research  focused  on  team  processes,  e.g., 
how  members  communicate,  what  they  communicate,  and  how  members  manage 
conflict.  Similarly,  we  found  a  large  body  of  research  on  how  to  train  people.  However, 
this  literature  was  largely  focused  on  the  individual  level,  although  a  smaller  and  more 
recent  line  of  research  has  focused  on  training  teams.  In  this  research,  much  of  the  focus 
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has  been  on  ways  to  increase  shared  understanding  of  “taskwork”  and  teamwork;  a  heavy 
focus  has  been  on  cross-training. 

There  have  been  notable  attempts  in  research  on  individual  level  training  to 
identify  contextual/situational  and  individual  factors  that  affect  training  outcomes.  The 
best  known  models  addressing  these  issues  are  provided  by  Quinones  (1995,  1997), 
Baldwin  and  Magjuka  (1997),  and  Mathieu  and  Martineau  (1997).  There  is  substantial 
overlap  between  the  models  in  the  identified  contextual/situational  and  individual  factors. 
Key  contextual/situational  factors  include  factors  in  the  organization  (e.g.,  climate, 
rewards,  goals,  situational  constraints,  participation,  management/organizational  support) 
and  in  groups  (e.g.,  group  composition,  cooperative  group  norms).  Key  individual 
factors  include  demographics  (e.g.,  age,  gender,  race),  motivation  (e.g.,  self-efficacy, 
expectancies),  knowledge,  skills,  and  abilities  developed  through  education  and  work 
experience,  personality  factors  (e.g.,  conscientiousness,  openness  to  experience, 
extraversion,  goal  orientation),  needs  (e.g.,  affiliation,  achievement,  dominance)  and 
work  attitudes  (e.g.,  job  involvement,  career  attitudes).  However,  clearly  other 
contextual/situational  or  individual  factors  could  also  play  a  role  in  training  outcomes, 
including  aspects  of  the  external  environment  (e.g.,  government  regulations,  competition, 
uncertainty),  organizational  structure  (e.g.,  formalization),  or  groups  (e.g.,  roles, 
including  conflict  or  ambiguity,  status,  size). 

More  recently,  researchers  have  attempted  to  identify  contextual/situational 
factors  affecting  team  training  outcomes.  Kozlowski  and  Salas  (1997)  provided  one  of 
the  most  comprehensive  models  addressing  these  issues,  identifying  techno-structural  and 
process  factors  at  the  organizational,  team,  and  individual  levels.  Techno-structural 
factors  translate  roughly  into  contextual/situational  factors,  particularly  those  factors  that 
relate  to  task  components  or  technology.  Process  factors  translate  roughly  into 
individual,  interpersonal/social,  and  organizational  factors  relating  to  interaction. 
Moreover,  Kozlowski  and  Salas  pointed  out  that  interactions  were  likely  to  occur  1) 
between  techno -structural  and  process  factors  within  one  level,  and  2)  between  factors  on 
different  levels.  They  discussed  these  interactions  using  the  term  congruence,  e.g., 
congruence  between  team  and  individual  level  goals. 
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Kozlowski  and  Salas  (1997)  further  noted  that  individual  differenee  factors 
(possess  by  team  members)  are  likely  to  play  a  role  in  the  effects  of  techno -structural  or 
process  factors  at  the  individual,  team,  and  organizational  levels.  In  brief,  at  the 
individual  level,  Kozlowski  and  Salas  identified  technical  skills  and  knowledge  (techno- 
structural  factors)  and  human  process  skills  and  knowledge  (process  factors).  At  the  team 
level,  they  identified  task  interdependence,  technology,  and  structure  (techno-structural 
factors)  as  well  as  teamwork,  leadership,  team  climate  and  team  coordination  (process 
factors). 

Finally,  at  the  organizational  level,  Kozlowski  and  Salas  (1997)  identified  goals, 
strategy,  resources,  technology  and  structure  (techno-structural  factors)  as  well  as  vision, 
rewards,  leadership,  organizational  culture/climate  (process  factors).  Thus,  the  Kozlowski 
and  Salas  model  seemed  to  provide  a  good  starting  point  for  identifying  socio-technical 
factors  that  could  be  addressed  as  part  of  the  TASL  program.  Also,  their  work  in 
conjunction  with  the  previously  mentioned  training  models  (e.g.,  Quinones,  1995,  1997) 
highlights  the  large  number  of  factors  that  could  play  a  role  in  training  for  collaborative 
computer-supported  work. 

So  from  a  literature  review  and  analysis  of  the  impact  of  collaborative  systems  on 
training,  two  overlapping  streams  of  research  seem  to  emerge.  The  first  stream  of 
research  focuses  primarily  on  training  at  the  individual  level  and  is  focused  on  the 
training  and  transfer  of  required  KSAs  (knowledge,  skills,  and  abilities),  and  relies  on 
both  lab  and  field  evaluation  approaches.  The  other  stream  of  research  would  focus  on 
teams — their  processes  (e.g.,  communication,  coordination)  and  effectiveness  (e.g., 
affective  outcome,  performance).  This  line  of  research  relies  on  the  case  study,  lab,  and 
field  approaches.  As  the  level  of  analysis  rises,  i.e.,  from  individual  to  team  to 
organization  level,  we  would  expect  to  see  corresponding  shifts  from  a  lab  experiment,  to 
field  study,  to  case  study  approach. 

2,1.3  Evaluation  and  Assessment 

The  area  of  evaluation  and  assessment  of  collaborative  systems  investigated 
relevant  research  in  military  and  nonmilitary  research  (emergency  operations,  business. 
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sociology,  software  design  and  medicine)  associated  with  methods,  metrics  and  tools  that 
could  potentially  support  an  evaluation  and  assessment  of  collaborative  systems  in 
military  logistics  operations  relevant  to  the  TASL  program.  We  discovered  that  current 
evaluative  methods  generally  fall  into  one  of  three  categories  to  include  1)  descriptive 
methods,  2)  assessment  methods,  and  3)  diagnostic  methods.  These  methods  are 
portrayed  in  Figure  5. 
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Figure  5,  Types  of  Evaluation  Methods 


Descriptive  methods  represent  a  first  step  in  evaluating  collaboration  and  serve  to 
“paint  a  picture”  of  the  collaborative  process.  These  methods  define  tasks  or  describe 
domains.  Examples  of  methods  in  this  category  are  cognitive  task  analysis,  ethnography, 
and  interviews  that  result  in  qualitative  descriptive  results.  Along  with  these  methods, 
come  data  reduction  techniques  and  representational  formalisms  that  take  the  often 
unwieldy  data  and  reduce  them  to  a  form  that  is  meaningful  and  succinct  by  highlighting 
trends,  patterns,  or  frequent  events.  Examples  of  this  kind  of  data  reduction  include 
concept  maps,  multidimensional  scaling,  and  social  networks.  The  end  product  of  these 
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methods  is  a  description  of  the  collaboration,  but  not  an  evaluation  of  it.  Assessment 
methods  take  the  descriptive  information  to  the  next  step  or  collect  new  information  that 
is  more  evaluative  in  nature.  This  step  usually  involves  comparing  a  description  or 
reduced  description  of  collaboration  to  an  expert  or  ideal  state.  For  example,  how  does 
the  collaboration  “stack  up”  to  one  known  to  be  highly  effective?  Expert  ratings  of 
collaboration  fall  into  this  category  as  do  evaluations  of  social  networks  by  quantitative 
comparison  to  some  standard  or  by  quantifying  choke  points  in  the  diagram,  for  instance. 
The  last  class  of  methods,  diagnostic  methods,  take  assessment  methods  one  step  further 
in  that  they  move  from  a  pure  assessment  of  the  collaboration  (e.g.,  good  vs.  poor,  90% 
vs.  50%)  to  a  richer  explanation  of  the  collaborative  behavior  underlying  the  assessment. 
So  for  example,  measures  that  go  deeper  than  effectiveness  to  focus  on  collaborative 
behaviors  such  as  situation  assessment,  sense  making,  conflict  management,  leadership, 
shared  mental  models,  etc.  and  that  tie  these  behaviors  to  the  assessment  fall  into  this 
category.  Note  that  just  describing  the  behaviors  is  not  diagnostic  until  a  connection  is 
made  to  the  assessment. 

In  this  literature  review,  most  of  the  methods  uncovered  could  be  classified  as 
descriptive,  though  a  few  involved  assessment.  There  are  virtually  no  measures  that  fall 
into  the  diagnostic  category,  though  there  is  some  research  that  is  beginning  to  investigate 
this  topic.  This  imbalance  is  a  reflection  of  the  state-of-the  art  when  it  comes  to 
understanding  and  evaluating  collaboration.  Diagnostic  measures  require  a  very  good 
understanding  of  what  good  collaboration  looks  like  within  a  domain  of  interest  and  how 
this  is  achieved.  Based  on  our  investigation  and  analysis,  both  of  these  knowledge  bases 
seem  to  be  lacking. 

Another  descriptive  method  worth  mentioning  involves  ethnography  (involving 
primarily  observation)  which  provides  largely  descriptive  data  on  collaboration.  One 
issue  associated  with  this  method  is  the  validity  of  the  interpretation  of  ethnographic  data, 
even  at  the  level  of  initial  observation.  Gerstl-Pepin  and  Gunzenhauser  (2002)  examined 
the  paradoxical  process  of  interpretation  in  collaborative  team  ethnography.  They  utilized 
examples  from  work  in  a  collaborative  team  evaluation  of  the  North  Carolina  A+  Schools 
Program.  They  found  conflicting  assumptions,  research  paradigms,  races,  genders,  class 
backgrounds,  and  research  interests.  They  examined  three  paradoxes:  interpretive 
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differences  on  the  research  team,  representation  of  diverse  voices  in  the  research  process, 
and  conflicting  roles  as  evaluators  and  critical  researchers.  Collaboration  led  to  a  greater 
understanding  through  multiple  meanings,  which  led  to  greater  fragmentation  and 
uncertainty.  Benyon,  Turner,  and  Turner  (2005)  discussed  the  use  of  ethnographic 
techniques  to  support  the  evaluation  of  cooperative  working  environments.  Their  study 
suggested  that  ethnographers  should  act  as  evaluators  of  early  concepts  or  prototype 
designs  before  requirements  are  finalized,  and  while  the  design  is  too  immature  to  benefit 
from  user  feedback. 

Based  on  our  analysis  of  evaluation  and  assessment  of  collaborative  systems,  we 
concluded  that  in  order  to  fully  evaluate  collaboration  within  a  specific  domain,  methods 
need  to  be  adapted  or  developed  for  all  levels  of  analysis.  There  are  a  number  of 
descriptive  methods  available  that  when  coupled  with  data  reduction  techniques,  can 
provide  good  summaries  of  the  collaborative  process.  What  is  primarily  needed  is  a 
mapping  of  these  types  of  descriptions  onto  a  measure  of  collaboration  effectiveness 
(e.g.,  expert  ratings,  comparison  to  referent)  and  then  a  mapping  to  collaboration 
behaviors  associated  with  effective  or  ineffective  outcomes.  Just  as  assessment  measures 
require  performance  criteria  for  comparison,  diagnostic  measures  require  behavioral 
criteria  for  comparison.  So  in  order  to  develop  evaluation  methods  that  assess  and 
diagnose  collaboration,  work  is  needed  to  develop  and  validate  measures  of  collaboration 
effectiveness  and  collaboration  behavior  such  as  situation  assessment,  leadership,  and 
shared  mental  models.  Descriptive  methods  that  might  be  applied  to  logistics 
collaboration  include  observations,  ethnography,  and  expert  interviews.  The  resulting 
data  can  be  further  analyzed  through  data  reduction  techniques  to  generate 
communication  or  social  networks.  As  metrics  for  collaboration  effectiveness  and 
collaboration  behaviors  evolve,  these  descriptions  can  be  associated  with  a  much  richer 
array  of  evaluative  information. 

2,1.4  Design  and  Visualization 

Our  review  and  analysis  in  the  area  of  design  and  visualization  was 
multidisciplinary  in  nature  and  focused  on  military  command  and  control  (C2)  systems 
while  aiming  at  a  collection  of  literature  that  has  specific  applicability  to  the  socio- 
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technical  aspects  of  Air  Force  (AF)  logistics  and  humanitarian  logistics.  We  focused 
particular  attention  on  military  C2  and  logistic  domains  that  involve  multiple  levels  of 
teamwork,  and  that  are  often  distributed  and  asynchronous  in  practice.  The  objective  of 
this  review  was  to  gain  a  comprehensive  understanding  and  to  pinpoint  multiple 
perspectives  of  what  has  been  referred  to  as  the  common  operational  picture  (COP).  The 
term  COP  is  typically  used  to  describe  a  visual  representation  of  aggregate  battle  space 
tactical,  operational,  and  strategic  information.  COP  is  an  information  tool  used  in 
command  control  centers  to  generate  situational  awareness  (Hager,  1997). 

Modern  joint  military  activities  employ  the  necessities  of  network-centric  warfare, 
complex  information  fusion,  and  emerging  interconnected  events  that  are  subject  to 
multiple  time  scales,  high  stakes  actions  taken  under  uncertainty  and  stress,  and  advanced 
sensors  that  deliver  a  glut  of  information  in  heterogeneous  representations  and  formats. 
Core  activities  involve  multiple  teams  working  together  to  pursue  a  given  purpose, 
mission,  or  task.  Unfortunately,  owing  to  the  tremendous  demands  and  pressures  of 
operations,  human  teamwork  and  operational  readiness  are  often  overcome  by  events  that 
lead  to  information  overload,  gaps  in  knowledge  sharing/congruity,  inadequate 
information  seeking  and  sharing,  task  saturation,  breakdowns  in  attention, 
miscommunications,  impending  failures,  and  miscalculations  experienced  during  both 
routine  and  non-routine  activities. 

Contemporary  patterns  of  military  decision-making  require  reassessment  of 
traditional  perspectives  to  enable  cognitive  readiness  and  agile  operations  that  correspond 
to  dynamic  threat  situations  of  the  2U'  century.  As  an  example  of  the  way 
interdisciplinary  collaboration  has  evolved,  multiple  teams/organizations  encounter 
situations  of  uncertainty  within  highly  vulnerable  -  even  volatile  -  environments,  but  are 
now  armed  with  the  capability  of  fusing  information  through  technologies  that  promote 
“anytime,  anywhere”  mobile  teamwork.  These  encounters  are  often  unexpected,  require 
joint  construction  of  knowledge  among  distributed  forces,  and  necessitate  perception  of 
an  amorphous  and  equally  distributed  threat  force.  Previously,  individual  analysts  and 
tactical  units  (i.e.,  teams)  have  not  had  the  broad  bandwidth,  information  retrieval-search 
capabilities  wherein  both  national-level  and  local-level  intelligence  can  be  brought  to 
bear  upon  a  source  problem  in  real  time.  This  high  information  profile  for  a  situation  -  at 
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first  pause  -  may  seem  to  provide  a  decisive  tactical  advantage.  However,  upon  closer 
reflection,  what  often  occurs  is  that  individual  analysts  may  be  overwhelmed  with  excess 
information  that  causes  a  person  to  engage  in  multiple  cognitive  processes  to  search  and 
cycle  through  what  is  relevant  based  on  what  they  are  experiencing.  For  practical  work 
this  means  that  individual  workers  are  often  tied  to  their  teamwork  by  the  levels  of 
information  they  encounter,  and  concurrently,  by  how  these  levels  of  information  are 
interrelated  to  the  group  process  or  product.  Because  these  transactive,  emergent 
elements  of  teamwork  are  often  mediated  by  information  and  communication 
technologies,  it  is  critical  that  intelligent  interfaces  bring  teams  together  to  1)  produce 
high  situational  awareness,  2)  reduce  information  overload,  and  3)  effectively  allocate 
functions/roles  to  enhance  cooperative  work.  These  formative  conditions  of  work  have 
been  the  foundation  of  which  the  statement  of  need  for  a  COP  has  evolved. 

There  is  much  to  be  explored  and  learned  from  COP  failures  and  successes  given 
the  differing  forms  of  implementation  in  real  environments.  However,  it  is  clear  that 
most  COP  artifacts  are  not  very  user-centric,  not  typically  designed  through  participation 
with  active  teams,  rarely  consider  the  ecological  or  contextual  perturbations  that  require 
adaptive  activities,  generally  ignore  lessons  learned  from  ethnographic  data,  and  are  not 
particularly  informed  from  current  CSCW  research.  When  considering  all  these 
shortcomings,  it  becomes  clear  that  COP  is  predominantly  a  techno-centric  enterprise  that 
results  in  some  form  of  artifact.  Although  more  experimental  studies  have  begun,  the 
area  is  woefully  underdeveloped  from  a  research  perspective.  In  summary,  there  is  much 
opportunity  for  future  research  evolving  from  the  TASL  program  to  explore  both  the 
concept  and  artifact  viewpoints  using  fieldwork  as  well  as  experimental  lab  research. 

2.2  Framework  for  Evaluation  of  Collaboration  and  Teamwork 

Traditional  approaches  to  the  design  and  development  of  information  systems 
have  concentrated  on  the  delivery  of  technology  rather  than  emphasizing  the  human  and 
organizational  characteristics  (Iqbal,  Gatward,  and  James  2005).  The  design  and 
development  of  collaborative  systems  and  tools  have  also  been  approached  in  the  same 
manner,  with  little  if  any  regard  given  to  cognitive,  social,  cultural,  and  organizational 
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impacts.  Many  collaborative  tools  and  systems  exist,  but  many  fail  (with  respeet  to  being 
aetually  fielded  or  fully  utilized)  due  to  inadequate  evaluation  of  user  requirements, 
produet  development,  and  implementation.  While  there  appears  to  be  fairly  widespread 
reeognition  of  the  impaet  of  the  soeio-teehnieal  influenees  associated  with  eollaborative 
systems,  there  is  little  if  any  researeh  related  to  understanding  how  to  better  address  these 
influences  (individually  and  collectively)  to  better  inform  the  design,  development,  and 
evaluation  of  eollaborative  systems.  One  of  the  more  signifieant  problems  is  that  metrics 
and  models  to  evaluate  eollaborative  tools  are  laeking,  therefore  indieating  that  what  is 
needed  (at  least  in  part)  is  a  framework  for  examining  soeio-teehnieal  issues  as  they 
pertain  to  eollaboration.  Developing  a  framework  for  evaluation  of  eollaboration 
technologies  is  an  ambitious  undertaking.  However,  as  technologies  and  tools  supporting 
eollaboration  eontinue  to  evolve,  and  the  nature  and  types  of  work  aetivities  and 
environments  (or  settings)  they  support  become  inereasingly  more  eomplex  to 
understand,  the  need  for  an  evaluation  framework  has  beeome  ever  more  crueial. 
Therefore,  it  was  deeided  that  making  some  progress  towards  the  development  and 
refinement  of  a  framework  for  assessing  eollaboration  was  another  logieal  veetor  of 
researeh  to  pursue  under  the  TASL  program.  The  foeus  of  this  researeh  was  twofold  - 
first  to  investigate  what,  if  any,  frameworks  or  models  eurrently  existed  for  assessing  or 
evaluating  eollaboration  or  eollaborative  systems,  and  second,  to  develop  and  assess  the 
utility  of  a  prototype  framework  that  eould  support  the  evaluation  of  soeio-teehnieal 
factors  in  collaborative  environments. 

In  researehing  the  literature  related  to  eurrent  models  or  frameworks  for 
evaluating  eollaboration,  Polivka’s  (1995)  eoneeptual  model  for  interagency 
collaboration  (see  Figure  6)  was  perhaps  the  most  promising  framework  found.  In  this 
model,  Polivka  identifies  three  eategories  of  input  to  the  eollaboration  environment: 
Environmental  Faetors,  Situational  Faetors,  and  Task  Charaeteristies.  These  eategories 
are  eonsistent  with  the  eonditions  associated  with  many  eollaborative  domains,  including 
the  military.  For  example,  it  is  important  that  ageneies  are  aware  of  the  goals  and 
eapabilities  of  other  ageneies  in  the  eollaborative  environment,  the  complexity  of  the  task 
must  be  well  understood,  and  the  soeial  priorities  must  be  well  artieulated.  These 
elements  are  routinely  eaptured  in  ease  studies  and  lessons  learned,  and  are  often 
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incorporated  into  training  programs,  formal  exercises,  and  standard  operating  proeedures. 
As  collaborations  begin  to  unfold,  Polivka’s  model  identifies  several  transaetional  faetors 
that  affect  the  means  by  which  agencies  interact  with  one  another.  These  transactional 
factors  tend  to  be  somewhat  dynamic  and  are  adapted  in  response  to  environmental 
factors,  situational  factors,  and  task  characteristics.  For  example,  as  the  resource 
channels  become  strained,  the  intensity  of  the  collaboration  increases.  In  emergeney 
response,  eomparisons  between  effective  resource  utilization  for  a  tornado  (MeEntire, 
2002)  and  resource  utilization  for  a  major  hurrieane  (e.g.,  Katrina  in  2005)  are  quite 
different.  The  magnitude  of  resources  required  for  hurricane  Katrina  were  massive  in 
comparison  with  the  tornado  relief  efforts  in  Ft.  Worth,  Texas,  which  was  reflected  in  the 
intensity  of  the  respective  response  efforts  and  the  eomplexity  of  the  collaborations. 


Figure  6,  Polivka’s  Model  for  Interagency  Collaboration 

Another  transactional  factor  identified  by  Polivka  is  that  of  formalization.  The 
model  establishes  the  importanee  of  recognizing  both  the  formal  and  informal  agreements 
as  well  as  pre-existing  rules  in  interagency  collaboration.  Agencies  prepared  in  this  way 
have  better  success  when  ehallenges  arise.  For  example,  when  formal  procedures  are  not 
appropriate,  teams  that  can  readily  adapt  informal  procedures  are  able  to  progress  more 
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rapidly.  Symon,  Long,  and  Ellis  (1996)  specifically  address  the  importance  of 
recognizing  both  formal  and  informal  exchanges  between  agencies  in  health  care  settings. 
In  their  examples,  they  identify  situations  in  which  unexpected  events  caused  formal 
procedures  to  break  down  and  how  various  agencies  successfully  and  unsuccessfully 
addressed  these  transitional  exchanges  with  informal  methods  and  procedures. 

Polivka  identifies  three  types  of  decision-making:  Mutual  Adjustment,  Corporate, 
and  Alliance.  Mutual  Adjustment  decision-making  is  where  there  are  only  a  few 
common  goals  between  agencies  and  a  few  individuals  adjust  to  accommodate  those 
goals  or  specific  situations.  Power  is  decentralized  in  this  method.  Corporate  decision¬ 
making  is  more  hierarchical  and  authoritarian.  With  corporate  decision  making,  formal 
rules  are  established  and  power  is  very  centralized.  Alliance  decision-making  is  a  hybrid 
of  both  Mutual  Adjustment  and  Alliance  decision-making  with  two  sub-categories  that 
include:  1)  federations  and  2)  coalitions  and  councils.  Federations  consist  primarily  of 
administrators  who  carry  out  the  actions  as  directed  by  unit  level  decision  makers.  With 
coalitions  and  councils  there  are  no  administrators;  decision  makers  are  the  professionals 
making  decisions  at  the  unit  level.  An  important  distinction  in  these  bodies  of  decision 
makers  is  that  with  Corporate  decision-making  decisions  are  from  the  top  down.  With 
Federation-type  Alliances,  the  decisions  are  made  from  the  bottom  up  (i.e.,  at  the  unit 
level  up  to  the  administrators). 

Although  Polivka’ s  conceptual  model  was  developed  in  a  domain  quite  far 
removed  from  military  logistics  (i.e.,  public  health),  this  foundational  work  seemed  to 
serve  as  a  useful  starting  point  for  considering  cross-agency  collaboration  across  a  range 
of  domains.  This  is  true  even  though  specific  elements  within  the  boxes  of  Figure  6  may 
change  in  importance  across  domains  or  even  scenarios  within  a  specific  domain  of 
interest  to  the  TASF  program.  In  discussing  the  use  of  Polivka’s  framework  during  the 
third  and  final  workshop,  it  was  concluded  that  while  the  framework  might  prove  useful 
in  attempts  to  describe  or  assess  collaboration,  as  our  understanding  of  collaboration  and 
how  to  assess  it  deepened,  this  framework  would  provide  little  guidance  in  terms  of 
diagnosing  collaborative  processes.  Therefore,  further  research  was  undertaken  under  the 
TASF  program  to  develop  a  framework  suitable  for  assessing  collaborative  technologies 
from  a  human  and  organizational  perspective,  recognizing  that  simply  testing 
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collaborative  technologies  and  systems  (hardware/software)  from  a  reliability  and 
network  connectivity  standpoint  does  not  guarantee  successful  implementation. 

The  prototype  framework  developed  and  evaluated  in  part  under  the  TASL 
program  is  portrayed  in  Figure  7.  It  is  a  unified  framework  intended  to  allow  researchers 
to  examine  factors  that  influence  collaboration  within  socio-technical  systems.  Such  an 
approach  addresses  the  impact  of  collaborative  technologies  from  a  systems’  perspective 
by  examining  factors  at  the  individual,  team,  and  organizational  levels  (Ritter,  Lyons,  and 
Swindler  2007).  The  present  model  integrates  methods  from  psychology,  sociology,  and 
anthropology  into  a  comprehensive  framework  to  assess  and  evaluate  barriers  to 
collaboration  in  organizations  from  a  human-centric  perspective.  Such  an  approach 
addresses  the  impact  of  collaborative  technologies  from  a  systems’  perspective  by 
examining  factors  at  the  individual,  team,  and  organizational  levels.  The  prototype 
model  is  not  intended  to  consider  all  barriers  to  collaboration  at  this  point.  For  purposes 
of  the  TASL  program,  we  included  barriers  thought  to  be  relevant  to  collaboration 
between  logistics  teams.  These  barriers  include:  information  loss,  reduced  trust,  reduced 
cohesion,  formation  of  local  coalitions,  and  suppressed  leadership  emergence  (see  Ritter 
et  ah,  2007  for  a  more  detailed  discussion  of  these  barriers). 


Figure  7,  Prototype  Framework  for  Assessing  Collaboration 
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There  are  four  phases  outlined  in  the  prototype  framework  depicted  in  Figure  7 
including:  Phase  I  -  Socio-Technical  Readiness  Assessment;  Phase  II  -  Domain  Survey; 
Phase  III  -  Directed  Assessment;  and  Phase  IV  -  Analysis.  The  primary  goal  of  Phase  I 
is  to  create  a  high-level,  profile  or  perspective  of  the  “work  system”  that  addresses  the 
organization,  people,  processes,  and  technology.  The  basic  product  resulting  from  this 
phase  includes  a  general  description  of  the  environment  (organizational  mission, 
products/services,  high-level  overview  of  business/work  processes,  business  systems, 
etc.).  The  methods  employed  in  gathering  this  information  would  include  survey 
instruments,  individual  interviews,  focus  group  discussions,  and  possibly  nominal  group 
techniques.  In  Phase  II,  Domain  Survey,  the  emphasis  is  on  obtaining  a  detailed 
understanding  of  the  domain,  including  a  more  in-depth  understanding  of  the  nature  of 
the  work,  information  flows,  communication  and  coordination  of  work  activities,  and 
potential  barriers  (and  facilitators)  to  collaboration  that  exist  within  the  organization. 
Methods  for  acquiring  information  and  data  in  this  phase  include  ethnography  (i.e.,  a 
qualitative  description  of  a  group,  organization,  or  culture),  descriptive  questionnaires, 
more  detailed,  structured  interviews,  and  possibly  social  networking  techniques  to  obtain 
insight  on  patterns  of  interaction  between  people/functions  within  the  organization.  The 
end  product  of  this  phase  is  a  more  detailed  description  of  the  organizational  profile 
characterizing  the  nature  of  collaboration  breakdowns  and/or  collaborative  successes  in 
the  team  or  organizational  context.  In  Phase  III,  Directed  Assessment,  the  objective  is  to 
attempt  to  quantitatively  measure  the  barriers  and/or  facilitators  identified  in  the  Phase  II. 
There  are  three  primary  methods  that  can  be  applied  in  this  phase  including  guided 
observations,  questionnaire -based  assessments,  and  performance  data.  The  observations 
in  this  phase  will  be  directed  toward  some  phenomenon  of  particular  interest  and  coded  to 
provide  quantitative  data.  For  example,  if  reduced  trust  was  identified  as  a  barrier  to 
successful  team  collaboration  in  Phase  II,  then  behavioral  indicators  of  trust,  such  as 
whether  or  not  a  team  member  rechecked  another  team  members’  work,  could  be  used  to 
measure  the  level  of  trust  between  team  members.  The  analysis  and  results  of  this  phase 
are  intended  to  produce  a  set  of  quantitative  data  based  on  the  collaboration  metrics  of 
interest  for  the  particular  domain  under  study. 
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The  collaborative  framework  described  is  intended  to  be  flexible  enough  to 
accommodate  the  unique  needs  of  various  domains  in  addition  to  logistics.  By  using  this 
framework  to  evaluate  the  utility  of  current  methods  and  tools  to  assess  collaboration,  and 
support  research  leading  to  the  refinement  of  the  same  (or  development  of  new 
methods/tools),  we  can  selectively  apply  methods  and  tools  to  meet  the  unique  demands 
encountered  in  various  domains.  Specifically,  we  see  this  framework  as  being  useful  to 
supporting  the  following: 

•  Assessing  collaboration  for  existing  systems/teams  in  organizations 

•  Comparing/contrasting  the  effects  of  various  organizational  factors  on 
collaboration 

•  Evaluating  the  impact  of  various  tools  on  team  collaboration 

•  Informing  and  supporting  the  design  of  collaborative  tools  or  systems  by 
providing  a  framework  for  assessing  the  human  (team)  aspects  of  collaboration 

The  development  of  the  prototype  framework  for  assessing  collaboration  discussed 
above  spawned  the  design  and  undertaking  of  two  notable  experiment  and  field 
evaluation  efforts  that  were  intended  to  provide  an  opportunity  to  begin  to  examine  and 
evaluate  our  proposed  framework  in  more  detail. 

The  experimental  effort  was  titled  the  “Computer-Based  Aerial  Port  Simulation” 
(CAPS).  The  objectives  for  CAPS  were:  1)  to  develop  a  computer-based  laboratory  task 
that  would  require  collaboration  within  a  distributed  logistics  network;  and  2)  to  examine 
social  and  psychological  factors  that  influence  team  performance  and  collaboration 
within  computer-based  distributed  teams.  CAPS  is  an  interactive,  computer-based 
simulation  program  that  models  specific  tasks  or  activities  accomplished  by  multiple 
logistic  functions  at  an  aerial  port  (Air  Force,  Air  Mobility  Command  organization).  The 
aerial  port  functions  modeled  include  the  Air  Terminal  Operations  Flight  (ATOF),  Ramp 
Services,  Passenger  Services,  Fleet  Services,  and  Cargo  Services.  These  functions,  as 
well  as  the  specific  team  and  individual  factors  to  be  evaluated  as  part  of  the  CAPS 
experiment,  are  shown  in  Figure  8.  The  design  and  development  of  the  CAPS  software 
also  incorporated  computer-mediated  communication  capabilities  such  as  a  “chat”  to 
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support  experiment  activities;  specifically  the  coordination  of  tasks  performed  by 
experiment  subjects  (see  Lyons,  Seyba,  and  Ames  2006  for  more  details). 
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Figure  8,  High-Level  CAPS  Experiment  Architecture 

The  CAPS  experiment  was  intended  to  support  an  examination  of  social  and 
psychological  factors  affecting  individual  and  team  performance,  and  collaboration 
within  a  computer-based  distributed  team  context.  The  CAPS  experiment  platform 
continues  to  support  other  AFRL/RHAL  sponsored  and  in-house  research  efforts, 
including  the  “Experimental  Evaluation  of  Collaborating  Teams”  research  task  being 
accomplished  under  a  separate  TAGS  delivery  order. 

In  addition  to  the  CAPS  experiment  effort,  AFRL/RHAL  was  also  successful  in 
establishing  a  relationship  with  the  Air  Mobility  Command,  Tanker  Airlift  Control  Center 
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(TACC)  at  Scott  AFB,  IL  to  obtain  support  for  collecting  data  and  information  in  support 
of  the  TASL  researeh  program.  The  intent  here  was  to  eonduet  a  domain  analysis  in  a 
representative  logistics  environment  that  would  allow  us  to  apply  and  assess  methods 
assoeiated  with  our  framework  for  evaluating  eollaboration.  This  field  study  was  also 
intended  to  directly  support  the  TACC  leadership  in  better  understanding  their  work 
proeesses  and  information  flows  as  part  of  planning  activities  for  the  US  Transportation 
Command  Fusion  Center  initiative. 

The  TACC  represents  a  eommand  and  eontrol  environment  responsible  for 
planning  and  exeeuting  airlift  and  tanker  missions  as  the  Air  Component  for  the  United 
States  Transportation  Command  (USTRANSCOM  (Distribution  Proeess  Owner  for  the 
Department  of  Defense)).  These  missions  primarily  include:  1)  the  movement  of  forees, 
equipment,  and  supplies  to/from  locations  worldwide,  2)  aeromedieal  evacuation,  and  3) 
air  refueling.  The  research  accomplished  in  part  under  the  TASL  program  with  respeet 
to  the  TACC  ineluded  a  domain  analysis  that  was  focused  on  doeumenting  the  business 
proeesses,  information  flows,  and  organizational  interaetions  (ineluding  external)  of 
planning  and  exeeution  funetions  within  the  TACC.  The  methods  used  to  eonduet  this 
analysis  ineluded  individual  interviews,  observations,  foeus  group  discussions,  and 
survey  questions.  The  survey  questions  were  developed  by  AFRL/RHAL  researehers  in 
support  of  an  overall  organizational  and  cultural  assessment  that  was  undertaken  as  part 
of  Air  Foree  Smart  Operations  for  the  2U'  Century  program  (AFS021)  efforts  at  AMC 
(separately  eontraeted  effort  by  AMC  with  Mainstream  GS  LLC).  The  questions 
developed  by  AFRL/RHAL  and  ineluded  as  part  of  this  assessment  focused  on  assessing 
unit  eohesion  between  members  of  funetions  within  the  TACC,  as  well  as  organizational 
and  eollaboration  pereeptions.  The  analysis,  results  and  recommendations  from  the 
AFRL/RHAL  researeh  assoeiated  with  this  assessment  are  diseussed  in  a  separate 
teehnieal  report  titled  ‘TACC  Culture  Assessment:  Opportunities  and  Challenges  ”,  dated 
9  Jan  2007.  In  addition,  findings  and  reeommendations  developed  from  the  data 
eolleetion,  interviews,  and  analysis  of  TACC  business  proeess  were  doeumented  in  a 
separate  memo  to  AMC  and  TACC  staff  functions,  and  formally  presented  to  the 
TACC/CV  by  members  of  the  AFRL/RHAL  staff  partieipating  in  the  TASL  researeh 
program. 
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2.3  Organizational  Simulation 

The  third  vector  of  research  pursued  under  the  TASL  program  was  focused  on 
investigating  and  exploring  the  feasibility  and  utility  of  using  organizational  simulation 
methodologies  that  could  potentially  support  the  development  of  a  viable,  in-house 
(AFRL/RHAL)  capability  to  research  organizational  phenomena  using  computational 
methods,  specifically  simulations  of  organizational  activity  or  structuring.  This  capability 
might  help  better  inform  the  design  of  organizations  utilizing  collaborative  systems.  The 
domain  of  interest  in  this  research  was  targeted  toward  Air  Force  Command  and  Control 
(C2)  organizations,  primarily  organizations  involved  in  the  planning,  monitoring,  and 
execution  of  logistics  activities  (e.g.,  a  Logistics  Readiness  Center),  but  also  other  Air 
Force  and  Joint  C2  organizations  (e.g..  Air  Mobility  Command  (AMC)  Tanker  Airlift 
Control  Center  (TACC),  or  USTRANSCOM  Deployment  Distribution  Operations 
Center. 

Organizational  simulation  involves  the  computational  representations  of  people, 
their  behaviors  and  interactions  in  response  to  the  environment  and  each  other  (Rouse 
and  Boff,  2005).  Organizational  simulations  can  provide  a  powerful  framework  for  1) 
testing  and  advancing  theories  related  to  organizational  behavior,  individual  and  team 
cognition,  leadership,  organizational  constructs,  etc.  2)  supporting  training  and 
organizational  learning,  and  3)  constructing  situational  based  models  to  understand  the 
potential  impact  of  strategy,  policy,  and  procedural  changes,  as  well  as  the  introduction  of 
new  systems,  on  overall  organizational  performance  (and  other  factors).  The  Air  Force, 
as  well  as  other  DoD  Service  components  are  undertaking  numerous  “transformational” 
initiatives  (e.g.,  AFS021,  USTRANSCOM  Fusion  Center  initiative,  etc.)  aimed  at 
streamlining  and  improving  our  overall  defense  posture  to  effectively  respond  to  current 
and  future  threats.  These  initiatives  will  undoubtedly  impact  (to  some  extent)  current 
military  doctrine,  authority,  policy,  process,  and  organizational  structures. 

The  focus  of  the  preliminary  research  conducted  under  the  TASL  program  was  on 
investigating  current  organizational  simulation  models  or  systems  to  determine  their 
suitability  for  researching  the  impact  of  various  organizational  designs  or  structures  on 
individual,  team,  and  organizational  performance  (in  a  command  and  control  situational 
context).  To  this  end,  a  literature  search  was  conducted  to  investigate  existing  simulation 
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software  supporting  organizational  simulations,  as  well  speeific  researeh  related  to  the 
use  of  organizational  simulation  in  various  eontexts.  Numerous  simulation  packages 
were  identified  that  were  well  suited  for  modeling  business  processes  (particularly 
manufacturing,  service  oriented,  and  job  shop  processes),  but  very  few  that  were 
specifically  designed  to  model  organizational  constructs  and  variables  or  factors  such  as 
communication  and  information  flows,  skill  levels,  and  reporting  hierarchies.  The 
research  conducted  by  Ashworth  and  Carley  (2007)  served  as  a  good  baseline  for  gaining 
a  better  understanding  of  the  capabilities  and  limitations  of  existing  simulation  models  in 
addressing  the  impact  of  various  organizational  improvement  initiatives  involving 
strategy,  policy,  process,  and/or  system  changes  on  organizational  structures. 

The  two  most  promising  models  we  identified  through  our  literature  search  and 
investigation  included  SimVision®,  a  commercial  off-the-shelf  simulation  package  for 
modeling  project  oriented  organizations,  and  POW-ER  (Process,  Organization,  Work  for 
Edge  Research),  a  simulation  platform  supporting  the  computational  modeling  of 
organizations  developed  at  Stanford  Elniversity.  Both  the  SimVision®  and  POW-ER 
software  are  outgrowths  of  the  Virtual  Design  Team  (VDT)  research  at  Stanford 
University  through  the  Collaboratory  for  Research  on  Global  Projects  (CROP),  headed  up 
by  Dr.  Raymond  Eevitt.  Dr.  Eevitf  s  Virtual  Design  Team  (VDT)  research  group  has 
developed  new  organization  theory,  methodology,  and  computer  simulation  tools  to 
design  organizations  that  can  optimally  execute  complex,  fast-track,  projects  and 
programs.  The  POW-ER  software  is  primarily  used  to  support  continuing  VDT  research 
at  Stanford  (and  other  educational  institutions)  related  to  the  modeling  and  validation  of 
micro-level  social,  cultural,  and  cognitive  factors  in  the  context  of  models  of  project- 
oriented  organizations. 

Under  the  TASE  program,  AERL/RHAL  was  able  to  secure  a  license  for  the 
SimVision®  software  from  the  National  Aeronautical  Space  Administration  (NASA)  so 
we  could  do  a  preliminary  evaluation  to  see  if  it  might  be  suitable  as  a  platform  for 
supporting  future  in-house  research  in  the  area  of  organizational  simulation  beyond  the 
TASE  program.  In  addition,  select  members  of  the  TASE  team  participated  in  a 
SimVision®  training  session  to  learn,  in  more  detail,  the  process  for  constructing  and 
validating  SimVision®  models.  With  respect  to  POW-ER,  AERE/RHAL  has  launched  a 
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new  research  task  that  is  investigating  the  use  of  POW-ER  as  a  platform  to  support 
further  research  in  the  area  of  organizational  simulation,  with  the  intent  of  possibly  using 
POW-ER  to  model  and  validate  the  impact  of  general  theories  and  empirical  based 
models  related  to  leadership,  trust,  etc.  that  could  eventually  be  incorporated  within 
organizational  models  to  address  the  impact  on  organizational,  team,  and  individual  level 
performance.  The  primary  (but  not  exclusive)  contextual  domain  to  inform  research  in 
this  area  would  be  logistics  command  and  control  type  organizations  such  as  the  TACC 
or  organizations  within  USTRANSCOM  such  as  the  Deployment  Distribution  Operations 
Center  (DDOC)  that  are  undergoing  potential  re-organizations  due  to  transformation 
initiatives  and  manpower  reductions. 

3.0  Conclusions  and  Recommendations 

As  highlighted  earlier,  the  EASE  research  program  encompassed  three  primary 
research  vectors  to  include:  an  analysis  of  “human  centered”  research  areas  associated 
with  collaborative  systems;  the  development  of  a  prototype  framework  that  could  support 
the  assessment  and  evaluation  of  collaboration  and  teamwork  in  organizations;  and 
finally,  an  investigation  of  research  and  models  supporting  the  simulation  of 
organizations.  Although  the  research  associated  with  each  of  these  vectors  progressed 
independently  (for  the  most  part)  over  the  course  of  the  EASE  program,  they  proceeded 
with  the  common  objective  of  gaining  a  deeper  understanding  about  collaborative 
systems  and  collaborative  environments  from  a  socio-technical  perspective. 

One  significant  finding  from  our  research  and  “human-centered”  analysis  of 
collaborative  systems  was  that  although  significant  research  has  attempted  to  address 
socio-technical  issues  in  computer  science  journals,  the  emphasis  was  primarily  on  the 
technological  challenges  and  costs  associated  with  software  and  hardware  supporting 
computer-mediated  communication,  and  much  less  about  social,  cultural,  and 
organizational  factors  of  interest  to  the  EASE  program.  In  contrast,  the  psychological 
literature  focuses  extensively  on  individual,  social,  and  organizational  factors  in  teams 
but  less  on  how  these  factors  should  be  addressed  in  the  design,  development,  and 
evaluation  of  collaborative  systems.  Hence,  what  seems  to  be  needed  in  moving  forward 
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is  an  integration  of  these  broad  lines  of  researeh,  for  instance,  examining  the  effects  of 
communication  losses  resulting  from  using  collaborative  systems  and  technologies  for 
computer  mediated  communication  to  support  virtual/dispersed  teams.  Two  other  topics 
for  further  research  that  should  be  included  in  this  integrated  approach  include  training 
and  teams.  The  training  research  should  focus  initially  on  gaining  a  better  understanding 
of  the  knowledge,  skills,  and  abilities  required  to  effectively  use  collaborative  systems  - 
beginning  at  the  individual  level,  and  progressing  to  look  at  the  training  of  teams  and 
organizations.  The  research  related  to  teams  should  focus  specifically  on  gaining  a  better 
understanding  of  team  processes  in  representative  C2  domains  or  environments.  The 
purpose  of  this  research  would  be  twofold  -  to  inform  the  training  of  teams  working  with 
collaborative  systems,  as  well  as  to  support  the  development  of  metrics  to  help  better 
assess  the  effectiveness  of  communication  and  coordination  between  team  members.  The 
design  and  development  of  meaningful  metrics  to  evaluate  the  human  and  organizational 
impacts  of  collaborative  technologies  and  systems  in  C2  domains  will  be  challenging  and 
require  a  combination  of  both  traditional  experimental  methods  and  ethnographic 
techniques. 

The  development  of  the  prototype  framework  for  assessing  and  evaluating 
collaboration  presented  in  this  report  was  an  ambitious  but  worthwhile  undertaking.  As 
technologies  and  tools  supporting  collaboration  continue  to  evolve,  and  the  nature  and 
types  of  work  activities  and  environments  (or  settings)  they  support  become  increasingly 
more  complex  to  understand,  the  need  for  an  evaluation  framework  will  become  even 
more  crucial  to  helping  ensure  the  successful  design,  development,  and  implementation 
of  collaborative  systems.  The  initial  research  associated  with  the  prototype  framework 
has  advanced  our  understanding  of  these  complex  issues.  However,  we  realize  that  a 
more  multi-disciplinary  perspective  will  need  to  be  developed  and  brought  to  bear  to 
address  the  challenges  associated  with  designing,  developing  and  implementing 
collaborative  systems  from  a  socio-technical  perspective.  The  current  focus  on 
technological  capabilities  for  these  systems  is  simply  not  sufficient  as  our  prototype 
framework  suggests.  The  proposed  framework  will  help  provide  a  better  understanding 
of  collaboration  within  organizations,  and  thus  better  inform  the  design,  development  and 
implementation  of  collaborative  systems  supporting  these  organizations. 
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The  final  vector  of  research  undertaken  as  part  of  the  TASL  program  focused  on 
examining  the  domain  of  organizational  simulation.  This  is  a  fairly  rich  area  for  future 
investigation  considering  most  of  the  current  models  we  identified  and  evaluated  in  our 
research  (particularly  those  in  academic  settings)  were  still  very  experimental  in  nature, 
and  only  addressed  specific  aspects  of  organizational  design,  and  factors  impacting 
organizational  performance.  The  SimVision®  and  POW-ER  models  evolving  from  the 
VDT  research  at  Stanford  University  over  the  past  20  years  seem  to  show  the  most 
promise  for  supporting  future  research  in  the  area  of  organizational  simulation  in  military 
C2  environments.  The  interest  should  be  investigating  the  use  of  organizational 
simulation  techniques  to  inform  the  design  of  military  organizations  and  business 
processes  based  on  social,  cognitive,  and  cultural  factors.  For  instance,  we  need  to 
develop  a  better  understanding  of  how  factors  such  as  leadership,  trust,  etc.  impact 
communication  and  coordination,  as  well  as  individual  and  team  performance  under 
various  organizational  design  constructs  (e.g.,  centralized  versus  decentralized  lines  of 
authority).  Advancing  our  understanding  in  these  areas  could  help  better  inform 
decisions  made  regarding  organizational  realignments  not  only  in  military  logistics  C2 
environments  (e.g.,  TACC,  USTRANSCOM  Fusion  Center),  but  other  domains  as  well. 
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Appendix  A:  TASL  Research  Program  Workshop  Principal 
Investigators 


Dr,  Deb  Steele-Johnson 

Dr.  Steele-Johnson  is  an  Assoeiate  Professor  of  Psyehology  in  the  Department  of 
Psyehology  at  Wright  State  University.  Dr.  Steele-Johnson’s  research  interests  focus  on 
how  people  acquire  complex  skills,  and  factors  that  affect  that  process;  as  well  as  on 
how  feedback,  goals,  other  self-regulatory  processes,  and  personal  characteristics  (e.g., 
need  for  achievement,  goal  orientations)  affect  learning  and  performance  on  complex 
tasks.  This  research  has  implications  for  motivation  and  training.  As  a  member  of  the 
TASL  team.  Dr.  Steele-Johnson  will  provide  expertise  related  to  gaining  a  better 
understanding  of  how  collaborative  systems  change  the  way  we  train  people. 

Dr.  Val  Shalin 

Dr.  Shalin  is  an  Associate  Professor  of  Psychology  in  the  Department  of  Psychology  at 
Wright  State  University.  Dr.  Shalin’s  research  interests  include  human  workplace 
expertise  and  associated  empirical  and  analytic  methods  to  support  the  design  and  testing 
of  workplace  aiding  and  training  technology.  She  also  studies  cognition  in  workplace 
settings,  incorporating  both  experimental  methods  from  behavioral  science  and 
observational  methods  from  social  science.  As  a  member  of  the  TASL  team.  Dr.  Shalin 
will  provide  expertise  related  to  organizational  and  technology  barrier  to  collaborative 
work  as  it  pertains  to  distributed  teams. 

Dr.  Mike  McNeese 

Dr.  McNeese  is  a  Professor  of  Information  Sciences  and  Technology,  and  the  Director  of 
the  User  Science  and  Engineering  (USE)  Eab  at  The  Pennsylvania  State  University.  Dr. 
McNeese’s  research  interests  include  human  interaction  with  information  technology  in 
complex  environments,  particularly  collaborative  systems  that  bring  together  the 
confluences  of  cognition,  computation,  collaboration,  and  context  for  given  fields  of 
practice.  As  a  member  of  the  TASE  team.  Dr.  McNeese  will  provide  expertise  related  to 
the  design  of  advanced  human-computer  interfaces  to  support  team-based,  logistics 
planning  and  command  control  activities  in  distributed  environments. 

Dr.  Alan  MacEachern 

Dr.  MacEachern  is  a  Professor  of  Geography  and  Director  of  the  Geo  VISTA  Center  at 
The  Pennsylvania  State  University.  Dr.  MacEachern’s  primary  research  interests  include 
the  interaction  between  formalized  visual  and  digital  representations  inherent  in  maps  and 
geographic  information  systems;  and  human  mental  representation  of  space  and  space- 
time.  As  a  member  of  the  TASE  team.  Dr.  MacEachern  will  provide  expertise  related  to 
the  visualization  and  representation  of  geospatial  data  and  information  supporting 
logistics  planning  and  command  control  activities  in  distributed  environments. 
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Dr,  Nancy  Cooke 

Dr.  Cooke  is  a  Professor  of  Applied  Psyehology  at  Arizona  State  University’s 
Polyteehnie  Campus,  and  the  direetor  of  the  Cognitive  Engineering  Institute  (CERI).  Dr. 
Cooke’s  researeh  areas  of  interest  inelude;  eognitive  engineering  and  knowledge 
elieitation  with  an  emphasis  on  eognitive  task  analysis,  team  eognition,  team  situation 
awareness,  mental  models,  expertise,  human-eomputer  interaetion,  eommand-and-eontrol 
in  unmanned  aerial  vehieles  and  emergeney  response  systems.  Dr.  Naney  Cooke  is  a 
renowned  expert  in  distributed  team  eognition  and  the  evaluation  of  eollaborative 
systems.  As  a  member  of  the  EASE  team.  Dr.  Cooke  will  provide  expertise  related  to  the 
development  of  methods  and  metries  for  evaluating  team  eollaboration. 

Dr.  Waleed  Smari 

Dr.  Smari  is  an  Assoeiate  Professor  of  Eleetrieal  and  Computer  Engineering  at  the 
University  of  Dayton.  Dr.  Smart’s  primary  researeh  areas  of  interest  ineludes  the  design 
and  engineering  of  software  and  hardware  for  eollaborative  systems;  parallel  and 
distributed  proeessing  and  networking,  and  performanee  evaluation  of  eomputing 
systems.  As  a  member  of  the  EASE  team.  Dr.  Smari  will  provide  expertise  in  the 
arehiteeture  and  design  of  eomputer  hardware  and  software  applieations  related  to 
eomputer  supported  eooperative  work  (CSCW). 
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Appendix  B:  TASL  Proposed  Research  Areas  and  Questions 


AREA  1  -  ORGANIZATION  /  PROCESS  /  CULTURE 

a.  What  are  the  technology  and  or  organizational  roadblocks  to  distributed 
collaboration? 

b.  How  can  we  analyze  proposed  process  changes  for  collaborative  systems  (task 
coupling,  interdependencies)? 

c.  How  can  we  remove  barriers  to  cross-agency  collaboration  (including 
understanding  the  process  of  collaboration  within  logistics,  and  identifying  and  defining 
barriers  to  cross-agency  collaboration)? 

d.  What  is  the  impact  of  ad-hoc  teams  on  collaboration? 


References: 

V.  Shalin  and  D. Steele- Johnson,  “Dimensions  of  Organization  and  Technology 
Influences  on  Collaboration”,  Report  of  Literature  Review,  December  2005. 

L.  Militello  et  ah,  “TASL  Literature  Review  Summary:  Cross  Agency  Collaboration”, 
December  2005. 


AREA  2  -  TRAINING 

a.  How  does  distributed,  computer-supported  work  affect  the  nature/type  of  training? 

b.  What  is  the  role  played  by  social,  organizational,  and  technical  factors  in  effective 
training  in  collaborative  systems? 


Reference: 

D.  Steele -Johnson  and  V.  Shalin,  “Training:  How  Collaborative  Systems  Change  the 
Way  We  Train  People”,  Report  of  Literature  Review,  December  2005. 


AREA  3  -  EVALUATION  /  ASSESSMENT 

a.  How  can  we  assess  collaboration  in  logistics  planning  (individual,  group, 
organizational  performance)? 
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b.  How  do  we  evaluate  the  effeetiveness  of  collaborative  technologies  (human 
performance  and  evaluation  tools  for  assessing  collaborative  technologies)?  We  may 
want  to  collaborate  on  this  research  with  AFRL/IFSD. 


Reference; 

N.  Cooke  et  ah,  “TASL  Area  #3:  Evaluation/ Assessment  Literature  Review”,  December 
2005. 


AREA  4  -  DESIGN  /  VISUALIZATION 

a.  How  do  workers  construct  a  common  operational  picture  in  an  emerging  or 
dynamically  changing  environment  (team  situation  awareness)? 

b.  What  do  we  need  to  understand  about  the  spatial  and  temporal  aspects  of  emerging 
or  dynamically  changing  environments  for  members  of  team  to  work  effectively?  How 
can  this  effectively  be  distributed? 

c.  What  is  the  requisite  knowledge  required  for  a  knowledge  manager? 

■  How  do  you  coordinate  in  collaborative  environment? 

■  See  reference:  “Knowledge  Management  in  the  Intelligence  Enterprise” 

Good  book  applicable  to  the  question  posed. 

■  How  does  one  know  when  to  act? 

■  What  is  the  metacognition? 

d.  What  is  the  role  of  visual  displays  in  facilitating  collaboration? 


Reference: 

M.  McNeese,  “Understanding  the  Common  Operational  Picture  from  Near  and  Ear”, 
Report  of  Literature  Review,  December  2005. 
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Acronyms 


AMC 

CAPS 

COP 

CSCW 

DARPA 

DoD 

JFCOM 

STREAM 

STS 

TACC 

TACS 

TASL 


Air  Mobility  Command 

Computer-Based  Aerial  Port  Simulation 

Common  Operating  Picture 

Computer  Supported  Cooperative  Work 

Defense  Advanced  Research  Projects  Agency 

Department  of  Defense 

Joint  Forces  Command 

Socio-Technical  Readiness  Evaluation  and  Assessment  Model 

Socio-Technical  System 

Tanker  Airlift  Control  Center 

Technology  for  Agile  Combat  Support 

Team-Based  Assessment  of  Socio-Technical  Logistics 
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